Concatenate with OME-Zarr v0.5 and sharding #104
Conversation
|
As of 83cd243 converting a 282 GB dataset (325 GB decompressed) took 7 minutes on 2 nodes with 64 CPUs each. As a reference converting a 65 GB dataset (183 GB decompressed) takes 2 minutes on 16 CPUs when using thread parallelism. |
|
Ran into zarr-developers/zarr-python#3221. |
pyproject.toml
Outdated
| "scikit-learn", | ||
| ] | ||
|
|
||
| [project.optional-dependencies] |
There was a problem hiding this comment.
Is this any different than the one from PyPI?
There was a problem hiding this comment.
Good question - I'm guessing no? @tayllatheodoro may know better
| sbatch_filepath: str = None, | ||
| sbatch_filepath: str | None = None, | ||
| local: bool = False, | ||
| block: bool = False, |
There was a problem hiding this comment.
What was the motivation for including the block parameter? Was it useful during testing?
There was a problem hiding this comment.
When running locally there isn't a good way to check if the jobs (processes) have finished. It is also useful for testing.
|
Current plan is that this PR will be merged after czbiohub-sf/iohub#301, updating the iohub dependency to the main branch. |
There was a problem hiding this comment.
LGTM, I was able to run biahub concatenate -c rechunk.yml -o test.zarr -sb sbatch.sh on a dataset which hasn't been converted from OME-NGFF v0.4/Zarr V2 to OME-NGFF v0.5/Zarr V3 over here /hpc/projects/intracellular_dashboard/organelle_dynamics/rerun/2025_04_15_A549_H2B_CAAX_ZIKV_DENV/2-assemble/zarr-v3.
|
Blocked until we bump waveorder: |
|
Another blocker is |
4 participants
Needs czbiohub-sf/iohub#311
To be investigated: multiprocessing based parallelism is not compatible with the asyncio-based thread parallelism that zarr-python is designed for and appears to be a bit slower.